FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding

Large-scale cross-lingual language models (LM), such as mBERT, Unicoder and XLM, have achieved great success in representation learning. However, when applied to zero-shot transfer tasks, most existing methods use only single-language input for LM finetuning, without leveraging the intrinsic alignment between different languages that proves essential multilingual tasks. In this paper, we propose FILTER, an enhanced fusion method takes data XLM finetuning. Specifically, FILTER first encodes text source its translation target independently shallow layers, then performs cross-language extract knowledge intermediate finally further language-specific encoding. During inference, model makes predictions based on language. For simple tasks classification, translated shares same label shared becomes less accurate or even unavailable more complex question answering, NER POS tagging. To tackle issue, additional KL-divergence self-teaching loss training, auto-generated soft pseudo-labels Extensive experiments demonstrate achieves new state of art two challenging multi-task benchmarks, XTREME XGLUE.